A virtual environment is a computer based simulated reality that may depict real or fictitious entities, such as characters and objects, in a space in which the entities may interact with each other and where one or more external actors can interact with the objects in the space and/or other actors. Examples of a virtual environment (VEnv) may present visual, audio, haptic or other stimuli to an actor, e.g. a gamer or user, who, in turn, acts on (or interacts with) entities in the simulated reality through touch, gestures, or movement, or through equipment capable of translating human action into action on an entity inside the VEnv (such as a virtual reality (VR) headset). In some examples, an actor is represented within a VEnv as an avatar that may be perceived by other actors who are using the VEnv at the same time.
Other examples of VEnvs could include a space capable of hosting containers such as virtual machines that are capable of executing programs or performing functions. Another example could be a digital environment in which conditions exist for agents representing real or abstract entities to exist and interact with each other.
Virtual entities and virtual environments may be used to provide several different types of user experiences. These include, but are not limited to, simulations, models of the evolution of systems, and multiple forms of electronic games (such as contests, missions, development of a narrative, etc.). However, conventional systems and methods for generating these types of virtual experiences, while quite advanced when compared to prior generations, still have inherent limitations. This is because the conventional way of satisfying a desire for greater gaming complexity is to increase the capabilities of the hardware (such as by increasing processor clock speed, available RAM, improved memory management techniques, etc.) to allow a greater number (and complexity) of instructions to be executed. This means that in some cases available memory or bandwidth limitations (e.g., for multi-player games played by non-collocated players communicating over a communications network) may still have the effect of preventing certain types of entity behavior and/or environment characteristics. Similarly, conventional gaming or virtual entity/virtual environment systems do not deliver a satisfactory user experience when it is desired that the entities interact in ways that require some degree of autonomy or independent decision making. This is because the entities within the virtual environment are typically centrally defined and controlled, making independent actions more difficult to implement without harming the capabilities of other entities, such as by restricting resources that those entities may require.
As an example of a conventional architecture for implementing a virtual environment, FIG. 1 is a diagram illustrating an example of a system architecture for a conventional gaming or other similar system 1 for simulating a virtual environment. In this example, a three dimensional (3D) gaming system that generates a game 10 includes a game engine 20 that executes computer implemented instructions (i.e., software code) to generate and permit interactions with the game/environment. The game engine typically includes a physics engine for simulating the motion and state of objects in the game, as those are determined by fundamental principles of mechanics and motion. For example, Newtonian physics may be applied to the objects to determine their motion in interactions with a user, where that motion is based on a fictitious “mass” assigned to the object. Alternatively or in addition, a fluid dynamics model may be applied to the object if a fluid environment is simulated; this typically relies on data such as flow rate and fluid/environment viscosity parameters.
Because of data processing or other resource constraints, simplifications or approximations are often required when using the physics models. These reduce the precision of the simulation, but require less data processing and, therefore, the results may be obtained more quickly for use in real-time simulations. For example, collision detection is often applied to object interactions and the objects are simplified in a physical model, such as by use of a non-visible mesh or collision geometry; this is used to define the geometric shape of the object and is often different from a visual mesh that defines the appearance of the object. Additional examples of the physics or dynamics processing that may be applied to objects include force feedback, Brownian motion, finite element analysis, ragdoll physics, cartoon physics, vehicle dynamics, rigid body dynamics, and soft body dynamics, each of which place demands on the data processing and visualization display capabilities of the system.
Game engine 20 calls subroutines 42 and 44 in libraries 40 to perform the calculations on the physical models of the objects in the simulation to determine the motion and state of the objects within the environment. Game engine 20 and the subroutines 42 and 44 make systems calls to kernel 50 via system call interface 52. Kernel 50 provides operations management and an abstraction of hardware 70. Rendering engine 30 processes the state data for objects along with representational graphics data for the objects using 3D graphics libraries 46 in order to generate the graphics/images for a scene. The 3D graphics libraries 46, in this example, access a GPU (graphics processing unit), graphics RAM and a display controller via system call interface 52 of kernel 50.
In a typical example, a game user throws a ball in the simulated environment and a Newtonian physics subroutine is called to apply Newtonian physics to the physical model for the ball (e.g. a sphere with a known mass) based on the force vector imparted to the ball in order to determine its resulting velocity and trajectory. Once the trajectory is known, then a collision detection subroutine may be called to determine the interactions with other objects in the simulation. If it is determined that the ball object collides with a wall, for example, then, if the ball and the wall are sufficiently hard, a rigid body subroutine may be called to determine how the ball moves after encountering the wall. Note that physical characteristic are defined by a user or programmer for the virtual environment and the objects under simulation, e.g. size, mass, drag coefficients, resilience, or viscosity.
Additional subroutines may be applied to objects in the virtual environment under simulation. For example, a particle system subroutine may be applied that typically uses a large number of small graphical objects known as sprites to simulate chaotic systems, such as natural phenomena (e.g., smoke, fire or flowing water, chemical reactions, explosions, etc.). Typically, individual sprites or particles within the simulated environment are modelled using other elements of the physics simulation; for example, fluid dynamics models may be applied to each sprite object, where the number of particles that can be simulated is generally limited by the computing power of the hardware. Thus, for purposes of reasonable game play, explosions may need to be modelled as a small set of large particles, rather than as a more accurate simulation of a large number of fine particles if the computing power is insufficient to perform the necessary calculations within the time available (e.g., between video frames of a computer game); this naturally reduces the realism and precision of the simulation.
State and spatial data is also typically maintained for a virtual environment. This data may include the size and configuration of a virtual space, the position of objects within that space, the orientation of the objects, and the velocity/movement of the objects. The state may also include the position and orientation of a virtual camera, which typically represents the perspective of a user of the system and is often the perspective from which the graphics are rendered.
One example of state and spatial data is a scene graph that defines the logical and spatial representation of entities or objects in a graphical scene. A scene graph is often a collection of nodes in a graph or tree structure. A tree node typically has a single parent, but may have multiple child nodes, where the effect of a parent is applied to all of its child nodes. In this way, an operation performed on a group may be automatically propagated to its members. For instance, multiple shapes and objects may be grouped together to form a compound object that can be acted upon, e.g. moved, transformed or selected, in much the same manner as a single object. For example, a driver object may be related to a vehicle object through a scene graph so that the driver moves along with the vehicle in the scene.
Similar to scene graphs are Bounding Volume Hierarchies (BVHs) that may be used to define a spatial structure for collision detection and other purposes. A BVH is a tree of bounding volumes, such as spheres or boxes, which are sized to closely enclose an object. In a hierarchical BVH tree, each node has a defined volume that can contain the volumes of the nodes beneath it in the hierarchy. An entire graphical scene, for example, may be the root of a tree graphic whose volume encloses all of the volumes of the other nodes in the tree. This may be useful for simplifying collision detection because if an object's bounding volume does not intersect a moving object's bounding volume (i.e., there is no collision between the objects), then the objects below that node also don't intersect the moving object's bounding volume and may be eliminated from consideration as sources of further interactions (at least to a first approximation).
Spatial partitioning may also be utilized to divide the virtual space of a scene into regions that may be organized into a tree, such as a binary space partitioning (BSP) tree, and for organizing objects in a virtual scene. Storing objects in a space partitioning data structure facilitates some types of geometric queries, such as ray tracing in order to determine whether a ray intersects an object. For example, the structure of a BSP tree may store spatial information about the objects in a virtual scene, such as their ordering from front-to-back with respect to the virtual camera representing a viewer's perspective. It may also be used in scanline algorithms to eliminate polygons in a virtual scene that are out of the virtual camera's viewing frustum, which limits the number of polygons that must be processed during rendering. It may also be useful in collision detection for determining whether two objects are close to one another. Further, BSP trees are used by some 3D games to contain static geometry for a scene to correctly merge movable objects, such as doors and characters onto the background of a scene, and are often combined with depth buffering to manage depth coordinates in 3D graphics.
The graphical appearance of objects in a simulation is obtained from a renderer or rendering engine 30 that produces two-dimensional or 3D graphics based on the graphical characteristics defined by representational graphics data for each object. Examples of representational graphics data may include a 3D mesh that defines a shape of the object along with the width, length, depth, color, pattern, texture, surface scattering and shading, light sources and models for the object and its environment. The graphics engine 30 may be a graphics rendering pipeline that applies the object state information and the representational graphics data in a first stage to create a geometry of the graphics to be presented, which is then passed to a geometry stage.
The geometry stage computes a model of what is to be drawn, how it should be drawn, and where it should be drawn; the model is then transformed into a 3D view space where the virtual camera is positioned at the center or origin of the view space. In different implementations, this stage may be one pipeline stage or multiple stages divided into different functional groups. One or more light sources are usually applied in the process of transforming the model to illuminate the view space. Projection operations may be applied, such as orthographic and perspective projection. Orthographic projection is typically used to represent a 3D model in a two-dimensional space and reduces distortion by ensuring that parallel lines remain parallel even after transformation. Perspective projection is applied so that an object appears smaller when the object model position is distant from the virtual camera; i.e., the farther away the object, the smaller it appears (similar to the way distant objects appear smaller to the human eye). Clipping is typically applied to remove objects that are outside of a view box for the virtual camera and screen mapping determines the coordinates of objects that are within the view box. Finally, at a rasterization stage, the elements within the view box, including the lines that have been drawn and any models that have been transformed are translated into pixels or picture elements.
The game engine 20 and rendering engine 30 generally run on a processor board for a user's platform, such as a personal computer, game console, or supercomputer. The amount of processing power available in the hardware platform 70 generally determines the precision of the simulation that the platform can support. The processors may include multiple cores, such as a multi-core microprocessor, to provide additional processing power for the physics processing and/or graphics processing. Additional processors, such as a dedicated physics processing unit (PPU) or a graphics processing unit (GPU), may be provided for hardware acceleration of the physics calculations. A GPU is also typically provided for computer graphics and image processing to more rapidly manipulate and modify memory to create images in a frame buffer for output to a display.
As mentioned, a conventional virtual environment, such as a video game, is typically generated and controlled by computer code hosted on an actor's machine, e.g. a gaming console, or remotely on one or more computers connected through a network. The computer code governs the behavior of the entities in the virtual environment, such as how the entities are displayed and how they respond to interaction with actors or other entities. The behavior and complexity of the virtual entities is generally limited by the performance capabilities of the combined computer code, execution platform, and communications network(s). For example, the networks may be limited by technical constraints on bandwidth or the number of users, which can impact the behaviors and interactions that the virtual entities are capable of demonstrating in an effective manner.
Conventional virtual environments are also commonly unable to update the controlling computer code while it is being executed (i.e., it is running). Further, the behavior of the virtual entities is often limited to the set of behaviors defined by the controlling computer code. In addition, it is often difficult or impossible for virtual entities hosted on different platforms or defined by different sources to interact with one another or to form ad hoc connections with each other. The ability to implement security controls, external data access, customizable storage, and input and output involving arbitrary sources is also limited. An actor in such a virtual environment is also generally unable to specify computing resource requirements based on their operational needs, and the resources may have limited or no ability to scale with the number of users, virtual entities, or virtual environments.
Conventional virtual environments also lack the ability to enumerate and address the virtual entities that are contained within the virtual environment. This typically forces an actor/user to navigate through a virtual environment in order to locate a desired virtual entity, unless the actor is provided with specific information on where to find the virtual entity within the virtual environment.
These constraints limit the complexity of virtual entities and the interactions they are capable of, both with other entities and with a virtual environment. These limitations also restrict how the virtual environment may be used and whether and how virtual environments from different sources can interoperate, which makes it difficult to build extensible virtual environment networks that can scale efficiently.
Embodiments of the invention are directed toward solving these and other problems individually and collectively.